fa11c03 refactor: Expose UndoReadFromDisk in header (MarcoFalke)

Pull request description:

  It is not possible to calculate the fee of a non-mempool transaction in RPCs unless txindex is active or the prevtxs are passed in through the RPC.

  Fix that issue for confirmed txs by exposing `UndoReadFromDisk` in the header file.

  This pull is a requirement for
  * rpc: faster getblockstats using BlockUndo data #14802
  *  Index for BIP 157 block filters #14121
  * my local patches

Tree-SHA512: 859ea5f2dfb4feac612b50faeb0e2b6c07b83f1d983e519d7647a78058d85c0390fd09ec66b460ae7a4c3b273e81b0013ee9f4bb8dfba0c4782ffaa1fa453ea6

d20d756 rpc: faster getblockstats using BlockUndo data (Felix Weis)

Pull request description:

  Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for `-txindex`, and works for all non-pruned blocks.

  ```
  # 2018-11-25T16:36:19Z Bitcoin Core version v0.17.99.0-edc715240-dirty (release build)
  seq 550100 550200  0.00s user 0.00s system 62% cpu 0.004 total
  xargs -n1 src/bitcoin-cli getblockstats  0.21s user 0.19s system 17% cpu 2.302 total

  # 2018-11-25T16:39:17Z Bitcoin Core version v0.17.0 (release build)
  seq 550100 550200  0.00s user 0.00s system 87% cpu 0.002 total
  xargs -n1 src/bitcoin-cli getblockstats  0.24s user 0.22s system 0% cpu 3:19.42 total
  ```

ACKs for commit d20d75:
  MarcoFalke:
    re-utACK d20d756

Tree-SHA512: 5babc3eb8d2fee2cb23dc12f522656b80737a540cbf2b13390a8f388304c46c064cca76f896b46a6e2abae8cc582d28e1ab20dd4bb17ad6142f20630c2d30c54

Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for -txindex, and works for all non-pruned blocks.

Github-Pull: bitcoin#14802
Rebased-From: d20d756 (diff minimised)

Summary:
Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for -txindex, and works for all non-pruned blocks.

bitcoin/bitcoin@d20d756

---

Depends on D6304

Backport of Core [[bitcoin/bitcoin#14802 | PR14802]]

Test Plan:
  ninja
  ./test/functional/test_runner.py rpc_getblockstats --gen-test-data
  ninja check-all

run bitcoin-cli getblockstats `non-pruned-block-height` on a node without -txindex now works

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Subscribers: deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D6305

RPC: Improve performance of `getblockstats` & `getblock`

See merge request bitcoin-cash-node/bitcoin-cash-node!849

This description below taken from !849 as instructed (with minor cleanups).
Thanks to Calin Culianu for bringing it all together.

The authors of this merge request were:

Co-Authored-By: Calin Culianu <calin.culianu@gmail.com>
Co-Authored-By: Axel Gembe <derago@gmail.com>

For the included parts of backport of bitcoin/bitcoin#14802:
Co-Authored-By: Felix Weis <mail@felixweis.com>
Co-Authored-By: MarcoFalke <falke.marco@gmail.com>

Summary
-------

It was observed during the recent round of ScaleNet tests by @ago that
the `getblockstats` call is very slow.  This call is used by block
explorers and as such the extant block explorers were timing out when
issuing `getblockstats` on large ScaleNet blocks.

`getblockstats` has a bunch of problems:

- It keeps `cs_main` held the entire time, which is not necessary.
- This is particularly problematic because it must iterate over all the
  inputs in a block, reading their ancestor tx's from disk. This
  operation can take tens of seconds to minutes on very large 300k input
  ScaleNet blocks.
- During this time `cs_main` is held and **the entire node freezes**.
- What's more, it would potentially read the same tx multiple times from
  disk as it iterated over the inputs.
- It also had a bunch of un-optimized UniValue API usage (this is
  minor).

What's changed:

- Release `cs_main` as soon as we don't need it anymore. It's only
  needed to look-up the block. Once we have the `CBlockIndex`, we don't
  need it anymore.  All of the rest of the code was reviewed carefully and
  is lock-free.
- Don't call `GetTransaction()` because it grabs `cs_main` and also is very slow
  and redundant. We instead backported part of Core#14802, and we use the
  undo (.rev) file to get previous inputs in order to calculate fees.
- Optimized the UniValue usage to our new (faster, safer) API.
- Leverage NRVO/RVO in C++ by returning only 1 thing in 1 branch (this
  is a minor optimizaton compared to all the others).

In addition, we modified `getblock` to release the lock earlier (was MR !850):

- We refactored `GetBlockChecked` (called in only 2 places, `getblock` and `getblockstats`)
  out into 2 functions: `ThrowIfPrunedBlock` (requires `cs_main`) and the slower `ReadBlockChecked` 
  (does *not* require locks). We also added the new `ReadUndoChecked` for reading the inputs from 
  the undo file.

Sorry for having 2 commits in 1 MR but the refactor was necessary, since to do it right we
needed to touch `getblock`.


Test Plan
---------

- Basic: `ninja check check-functional`
- Advanced: We want to illustrate the speedup here, so you will need
  ScaleNet for best results.
  1. Synch up a ScaleNet node, be sure to enable RPC on it in the conf
     file or via the right CLI arg magic. Also **enable** `txindex=1`
     (you will need this only for testing the version of the client without this MR)
  2. Create a shell script similar to the below, call it `cmd.sh`:

```bash
#!/bin/bash

blocks="16029 16026 16037 16130 16179 15813 16124"

for ht in $blocks; do
    src/bitcoin-cli -conf=$HOME/bitcoin/bchn/scalenet.conf getblockstats $ht &
done

wait
```

 3. **Note:** Be sure to replace the `-conf` arg in the shell script above with
    the conf file you created for your scalenet node.
 4. Next, build `bitcoind` with this commit and start the node, wait for it to
    synch fully and settle.
 5. Execute the following: `time ./cmd.sh`  (if you get an error about
    txindex, this is normal and unrelated to this commit. Wait a bit for it to
    catch up and try again).
 6. You should see it spit out a bunch of JSON results and a time.
 7. Do 5-6 above again but this time with a `bitcoind` built against
    master (without this commit).  Note how the time is multiple times longer,
    or more, depending on your machine.

This commit enhance the node to no longer need `txindex` for `getblockstats`,
otherwise correctness of the `getblockstats` call is identical, and now more
efficient.

Special thanks to `Axel Gembe <derago@gmail.com>` for his hard work in
troubleshooting and investigating this.  This code is largely inspired
by his test code here: https://gitlab.com/ago/bitcoin-cash-node/-/tree/optimize_getblockstats

Also special thanks to `Mark Lundeberg <5505128-markblundeberg@users.noreply.gitlab.com>`
for his tip on using the latest core changes that use the undo file rather
than the (slower) txindex lookups per input.

…data

66d012a test: RPC: getblock fee calculations (Elliott Jin)
bf7d6e3 RPC: getblock: tx fee calculation for verbosity 2 via Undo data (Elliott Jin)

Pull request description:

  This change is progress towards bitcoin#18771 .  It adapts the fee calculation part of bitcoin#16083 and addresses some feedback.  The additional "verbosity level 3" features are planned for a future PR.

  **Original PR description:**

  > Using block undo data (like in bitcoin#14802) we can now show fee information for each transaction in a block without the need for additional -txindex and/or a ton of costly lookups. For a start we'll add transaction fee information to getblock verbosity level 2. This comes at a negligible speed penalty (<1%).

ACKs for top commit:
  luke-jr:
    tACK 66d012a
  fjahr:
    tACK 66d012a
  MarcoFalke:
    review ACK 66d012a 🗜

Tree-SHA512: be1fe4b866946a8dc36427f7dc72a20e10860e320a28fa49bc85bd2a93a0d699768179be29fa52e18b2ed8505d3ec272e586753ef2239b4230e0aefd233acaa2

d20d756 rpc: faster getblockstats using BlockUndo data (Felix Weis)

Pull request description:

  Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for `-txindex`, and works for all non-pruned blocks.

  ```
  # 2018-11-25T16:36:19Z Bitcoin Core version v0.17.99.0-edc715240-dirty (release build)
  seq 550100 550200  0.00s user 0.00s system 62% cpu 0.004 total
  xargs -n1 src/bitcoin-cli getblockstats  0.21s user 0.19s system 17% cpu 2.302 total

  # 2018-11-25T16:39:17Z Bitcoin Core version v0.17.0 (release build)
  seq 550100 550200  0.00s user 0.00s system 87% cpu 0.002 total
  xargs -n1 src/bitcoin-cli getblockstats  0.24s user 0.22s system 0% cpu 3:19.42 total
  ```

ACKs for commit d20d75:
  MarcoFalke:
    re-utACK d20d756

Tree-SHA512: 5babc3eb8d2fee2cb23dc12f522656b80737a540cbf2b13390a8f388304c46c064cca76f896b46a6e2abae8cc582d28e1ab20dd4bb17ad6142f20630c2d30c54

d20d756 rpc: faster getblockstats using BlockUndo data (Felix Weis)

Pull request description:

  Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for `-txindex`, and works for all non-pruned blocks.

  ```
  # 2018-11-25T16:36:19Z Bitcoin Core version v0.17.99.0-edc715240-dirty (release build)
  seq 550100 550200  0.00s user 0.00s system 62% cpu 0.004 total
  xargs -n1 src/bitcoin-cli getblockstats  0.21s user 0.19s system 17% cpu 2.302 total

  # 2018-11-25T16:39:17Z Bitcoin Core version v0.17.0 (release build)
  seq 550100 550200  0.00s user 0.00s system 87% cpu 0.002 total
  xargs -n1 src/bitcoin-cli getblockstats  0.24s user 0.22s system 0% cpu 3:19.42 total
  ```

ACKs for commit d20d75:
  MarcoFalke:
    re-utACK d20d756

Tree-SHA512: 5babc3eb8d2fee2cb23dc12f522656b80737a540cbf2b13390a8f388304c46c064cca76f896b46a6e2abae8cc582d28e1ab20dd4bb17ad6142f20630c2d30c54

d20d756 rpc: faster getblockstats using BlockUndo data (Felix Weis)

Pull request description:

  Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for `-txindex`, and works for all non-pruned blocks.

  ```
  # 2018-11-25T16:36:19Z Bitcoin Core version v0.17.99.0-edc715240-dirty (release build)
  seq 550100 550200  0.00s user 0.00s system 62% cpu 0.004 total
  xargs -n1 src/bitcoin-cli getblockstats  0.21s user 0.19s system 17% cpu 2.302 total

  # 2018-11-25T16:39:17Z Bitcoin Core version v0.17.0 (release build)
  seq 550100 550200  0.00s user 0.00s system 87% cpu 0.002 total
  xargs -n1 src/bitcoin-cli getblockstats  0.24s user 0.22s system 0% cpu 3:19.42 total
  ```

ACKs for commit d20d75:
  MarcoFalke:
    re-utACK d20d756

Tree-SHA512: 5babc3eb8d2fee2cb23dc12f522656b80737a540cbf2b13390a8f388304c46c064cca76f896b46a6e2abae8cc582d28e1ab20dd4bb17ad6142f20630c2d30c54

d20d756 rpc: faster getblockstats using BlockUndo data (Felix Weis)

Pull request description:

  Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for `-txindex`, and works for all non-pruned blocks.

  ```
  # 2018-11-25T16:36:19Z Bitcoin Core version v0.17.99.0-edc715240-dirty (release build)
  seq 550100 550200  0.00s user 0.00s system 62% cpu 0.004 total
  xargs -n1 src/bitcoin-cli getblockstats  0.21s user 0.19s system 17% cpu 2.302 total

  # 2018-11-25T16:39:17Z Bitcoin Core version v0.17.0 (release build)
  seq 550100 550200  0.00s user 0.00s system 87% cpu 0.002 total
  xargs -n1 src/bitcoin-cli getblockstats  0.24s user 0.22s system 0% cpu 3:19.42 total
  ```

ACKs for commit d20d75:
  MarcoFalke:
    re-utACK d20d756

Tree-SHA512: 5babc3eb8d2fee2cb23dc12f522656b80737a540cbf2b13390a8f388304c46c064cca76f896b46a6e2abae8cc582d28e1ab20dd4bb17ad6142f20630c2d30c54

d20d756 rpc: faster getblockstats using BlockUndo data (Felix Weis)

Pull request description:

  Using undo data for a block (rev?????.dat) we can retrieve value information about prevouts and calculate the final transaction fee (rate). This approach is about 80x faster, drops the requirement for `-txindex`, and works for all non-pruned blocks.

  ```
  # 2018-11-25T16:36:19Z Bitcoin Core version v0.17.99.0-edc715240-dirty (release build)
  seq 550100 550200  0.00s user 0.00s system 62% cpu 0.004 total
  xargs -n1 src/bitcoin-cli getblockstats  0.21s user 0.19s system 17% cpu 2.302 total

  # 2018-11-25T16:39:17Z Bitcoin Core version v0.17.0 (release build)
  seq 550100 550200  0.00s user 0.00s system 87% cpu 0.002 total
  xargs -n1 src/bitcoin-cli getblockstats  0.24s user 0.22s system 0% cpu 3:19.42 total
  ```

ACKs for commit d20d75:
  MarcoFalke:
    re-utACK d20d756

Tree-SHA512: 5babc3eb8d2fee2cb23dc12f522656b80737a540cbf2b13390a8f388304c46c064cca76f896b46a6e2abae8cc582d28e1ab20dd4bb17ad6142f20630c2d30c54